Skip to content

Add Prometheus metrics with PromEx - #275

Merged
ericmj merged 3 commits into
mainfrom
claude/metrics-dashboards-architecture-4b4lwj
Jul 26, 2026
Merged

ericmj merged 3 commits into
mainfrom
claude/metrics-dashboards-architecture-4b4lwj

Conversation

@ericmj

@ericmj ericmj commented Jul 26, 2026

Copy link
Copy Markdown
Member

Part of the monitoring rollout (see hexpm/hexpm-ops#218, which deploys the Prometheus + Grafana stack that scrapes this endpoint).

What this does

  • Adds prom_ex (resolves to 1.11.0 with the current ecto lock) with a new Bob.PromEx module: Application, Beam, and Phoenix plugins everywhere; the Ecto plugin only on the master, since agents run without Bob.Repo.
  • Adds a custom Bob.PromEx.Plugins.Bob plugin for the build system:
    • queue depth by job type (polls Bob.Queue.queue_sizes/0, master-only; emits zeroes for configured-but-empty queues so gauges don't go stale)
    • in-flight jobs per node (polls Bob.Runner.state/0)
    • job duration histogram with ok/error result labels (instrumented around the existing :timer.tc site in Bob.Runner.run_task/2)
    • timeout and crash counters on the runner's kill paths
    • labels use job_type (e.g. BuildOTP-amd64) to avoid colliding with Prometheus' built-in job label
  • Serves /metrics on a standalone Bandit listener (PromEx.Plug), enabled only when BOB_METRICS_PORT is set — production sets 9568 via the hexpm-ops manifests; dev listens on 9568; tests disable PromEx. The Hetzner/Scaleway agents don't set the variable, so nothing changes there (agent metrics are a planned follow-up).
  • PromEx starts after Bob.Repo, and queue polling tolerates the Repo being unavailable (rescues connection/registration errors) — this fixed a boot-order race found during a live /metrics smoke test.

Notes

  • Full suite green: 351 tests, 0 failures. Compiles with --warnings-as-errors, format check clean.
  • Verified end-to-end locally: booted the app against Postgres and curled the metrics endpoint (BEAM, Phoenix, and custom queue metrics all export).

🤖 Generated with Claude Code


Generated by Claude Code

Serve metrics on a standalone Bandit listener on BOB_METRICS_PORT with
the Application, Beam, and Phoenix PromEx plugins, plus a custom plugin
for the build queue: queue depth by job type, in-flight jobs, job
duration/result, timeouts, and crashes. The Ecto plugin and queue depth
polling only run on the master since agents have no database.

Co-Authored-By: Claude <noreply@anthropic.com>
@ericmj
ericmj marked this pull request as ready for review July 26, 2026 17:27
ericmj added 2 commits July 26, 2026 10:53
PromEx's Phoenix plugin attaches to [:phoenix, :endpoint, :stop], which
comes from Plug.Telemetry. The endpoint never installed it, so every
Phoenix dashboard panel stayed empty while the app served traffic.
Prometheus scrapes every 30s, so a 30s dashboard interval leaves one sample
per window and every irate/increase panel renders empty.
@ericmj
ericmj merged commit 4d08355 into main Jul 26, 2026
4 checks passed
@ericmj
ericmj deleted the claude/metrics-dashboards-architecture-4b4lwj branch July 26, 2026 21:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants